Automated system and method for modeling the behavior of vehicles and other agents

ABSTRACT

A method and apparatus are provided for determining one or more behavior models used by an autonomous vehicle to predict the behavior of detected objects. The autonomous vehicle may collect and record object behavior using one or more sensors. The autonomous vehicle may then communicate the recorded object behavior to a server operative to determine the behavior models. The server may determine the behavior models according to a given object classification, actions of interest performed by the object, and the object&#39;s perceived surroundings.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 15/278,338, filed Sep. 28, 2016, which is a continuation ofU.S. patent application Ser. No. 13/446,494, filed Apr. 13, 2012, nowU.S. Pat. No. 9,495,874, the disclosures of which are incorporatedherein by reference.

BACKGROUND

Autonomous vehicles use various computing systems to aid in thetransport passengers from one location to another. Some autonomousvehicles may require some initial input or continuous input from anoperator, such as a pilot, driver, or passenger. Other systems, forexample autopilot systems, may be used only when the system has beenengaged, which permits the operator to switch from a manual mode (wherethe operator exercises a high degree of control over the movement of thevehicle) to an autonomous mode (where the vehicle essentially drivesitself) to modes that lie somewhere in between.

BRIEF SUMMARY

An apparatus for managing behavior models used by an autonomous vehicleto predict the behavior of a corresponding object is disclosed. In oneaspect, a server determines behavior models to be used by an autonomousvehicle to predict the behavior of a detected object. The server mayinclude a memory configured to store first object data and behaviormodel data for a first object, and the server may include a processor incommunication with the memory. The processor being configured to receivesecond object data from a monitoring source, wherein the second objectdata comprises world-view data and actions-of-interest data for adetected object. The processor also being configured to analyze thesecond object data to determine the detected object's classification andto determine whether a behavior model exists for the detected objectbased on the detected object's classification. The processor furtherconfigured to create the behavior model for the detected object based onthe world-view data and actions-of-interest data, when a behavior modeldoes not exist for the detected object's classification. Alternatively,the server may update the behavior model for the detected object basedon the world-view data and actions-of-interest data, when the behaviormodel exists for the detected object's classification.

In another aspect, the second object data indicates that the detectedobject is in proximity to one or more nearby objects, and the world-viewdata comprises position and movement data of one or more of the nearbyobjects from a perspective of the detected object.

In yet another aspect, the actions-of-interest data indicates that thedetected object has been positioned or has moved in a predefined mannerrelative to one or more road graph elements. The road graph elements maycomprise a first lane of traffic and a second lane of traffic, and theactions-of-interest data may indicate that the detected object haschanged from the first lane of traffic to the second lane of traffic.

In still another aspect, the monitoring source is an autonomous vehicle,and the detected object is one of an automobile, a pedestrian, or abicycle.

In another aspect, the behavior model may comprise a plurality ofprobabilities, and at least one probability is based on a path of travelthat the first object was observed traveling. At least one probabilityof the plurality of probabilities may identify a probability that asecond object will travel an associated path of travel previouslytraveled by the first object. In addition, the behavior model maycomprise a plurality of probabilities, wherein at least one probabilityis based on first world-view data for the first object, and wherein atleast one probability is based on first actions-of-interest data for thefirst object.

In still another aspect, the server is further configured to determine aplurality of behavior models for the detected object's classification,and wherein each behavior model corresponds to a different set ofworld-view data or actions-of-interest data. The server is may also beconfigured to communicate the behavior model to an autonomous vehicle,which may alter its control strategy in response to the communicatedbehavior model.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a functional diagram of an autonomous navigation system inaccordance with an aspect of the disclosure.

FIG. 2 is an exemplary design of the interior of an autonomous vehiclein accordance with an aspect of the disclosure.

FIG. 3 is a view of the exterior of an exemplary vehicle in accordancewith an aspect of the disclosure.

FIGS. 4A-4D are views of the sensor fields for an autonomous vehicle.

FIG. 5 is a view of an autonomous vehicle traveling autonomously inproximity to other vehicles.

FIG. 6 is map in accordance with an aspect of the disclosure.

FIG. 7 is a functional diagram of a behavior model server in accordancewith an aspect of the disclosure.

FIG. 8 is a flow chart for a behavior server in accordance with anaspect of the disclosure.

FIG. 9 is a flow chart for an autonomous vehicle in accordance with anaspect of the disclosure.

FIG. 10 a functional diagram of a communication system in accordancewith an aspect of the disclosure.

DETAILED DESCRIPTION

Aspects of the disclosure relate generally to a system and method ofmodeling behavior of objects detected by an autonomous vehicle. Inparticular, a vehicle implementing the autonomous driving system iscapable of detecting and reacting to surrounding objects. Some of thedetected objects will be mobile, such as pedestrians, automobiles, andbicycles. As set forth below, the autonomous driving system is able toidentify surrounding objects and determine how those objects perceivetheir surroundings. The system is also able to identify when specificactions of interest have been performed by the detected object. Theautonomous vehicle records the data relating to the detected object'sperceived surroundings, as well as any identified action of interest,and then communicates the recorded data to a server for behaviormodeling. The server may then provide the autonomous vehicle withupdated modeling data, allowing the autonomous vehicle to predict theobject's likely movements. In turn, the vehicle may react to nearbyobjects in a way that decreases the likelihood of an accident andincreases the efficiency of travel.

As shown in FIG. 1, an autonomous driving system 100 in accordance withone aspect of the disclosure is included within a vehicle 101 withvarious components. While certain aspects of the disclosure areparticularly useful in connection with specific types of vehicles, thevehicle may be any type of vehicle including, but not limited to, cars,trucks, motorcycles, busses, boats, airplanes, helicopters, lawnmowers,recreational vehicles, amusement park vehicles, trams, golf carts,trains, and trolleys. The vehicle may have one or more computers, suchas computer 110 containing a processor 120, memory 130 and othercomponents typically present in general purpose computers.

The memory 130 stores information accessible by processor 120, includinginstructions 132 and data 134 that may be executed or otherwise used bythe processor 120. The memory 130 may be of any type capable of storinginformation accessible by the processor, including a computer-readablemedium, or other medium that stores data that may be read with the aidof an electronic device, such as a hard-drive, memory card, ROM, RAM,DVD or other optical disks, as well as other write-capable and read-onlymemories. Systems and methods may include different combinations of theforegoing, whereby different portions of the instructions and data arestored on different types of media.

The instructions 132 may be any set of instructions to be executeddirectly (such as machine code) or indirectly (such as scripts) by theprocessor. For example, the instructions may be stored as computer codeon the computer-readable medium. In that regard, the terms“instructions” and “programs” may be used interchangeably herein. Theinstructions may be stored in object code format for direct processingby the processor, or in any other computer language including scripts orcollections of independent source code modules that are interpreted ondemand or compiled in advance. Functions, methods and routines of theinstructions are explained in more detail below.

The data 134 may be retrieved, stored or modified by processor 120 inaccordance with the instructions 132. For instance, although the systemand method is not limited by any particular data structure, the data maybe stored in computer registers, in a relational database as a tablehaving a plurality of different fields and records, XML documents orflat files. The data may also be formatted in any computer-readableformat. By further way of example only, image data may be stored asbitmaps comprised of grids of pixels that are stored in accordance withformats that are compressed or uncompressed, lossless (e.g., BMP) orlossy (e.g., JPEG), and bitmap or vector-based (e.g., SVG), as well ascomputer instructions for drawing graphics. The data may comprise anyinformation sufficient to identify the relevant information, such asnumbers, descriptive text, proprietary codes, references to data storedin other areas of the same memory or different memories (including othernetwork locations) or information that is used by a function tocalculate the relevant data.

The processor 120 may be any conventional processor, includingcommercially available processors. Alternatively, the processor may be adedicated device such as an ASIC or FPGA. Although FIG. 1 functionallyillustrates the processor, memory, and other elements of computer 110 asbeing within the same block, it will be understood by those of ordinaryskill in the art that the processor and memory may actually comprisemultiple processors and memories that may or may not be stored withinthe same physical housing. For example, memory may be a hard drive orother storage media located in a housing different from that of computer110. Accordingly, references to a processor or computer will beunderstood to include references to a collection of processors orcomputers or memories that may or may not operate in parallel. Ratherthan using a single processor to perform the steps described herein someof the components such as steering components and decelerationcomponents may each have their own processor that only performscalculations related to the component's specific function.

In various of the aspects described herein, the processor may be locatedremote from the vehicle and communicate with the vehicle wirelessly. Inother aspects, some of the processes described herein are executed on aprocessor disposed within the vehicle and others by a remote processor,including taking the steps necessary to execute a single maneuver.

Computer 110 may include all of the components normally used inconnection with a computer, such as a central processing unit (CPU),memory (e.g., RAM and internal hard drives) storing data 134 andinstructions such as a web browser, an electronic display 140 (e.g., amonitor having a screen, a small LCD touch-screen 141 or any otherelectrical device that is operable to display information), user input(e.g., a mouse, keyboard, touch screen and/or microphone), as well asvarious sensors (e.g. a video camera) for gathering the explicit (e.g. agesture) or implicit (e.g. “the person is asleep”) information about thestates and desires of a person.

The vehicle may also include a geographic position component 144 incommunication with computer 110 for determining the geographic locationof the device. For example, the position component may include a GPSreceiver to determine the device's latitude, longitude and/or altitudeposition. Other location systems such as laser-based localizationsystems, inertial-aided GPS, or camera-based localization may also beused to identify the location of the vehicle. The location of thevehicle may include an absolute geographical location, such as latitude,longitude, and altitude as well as relative location information, suchas location relative to other cars immediately around it which can oftenbe determined with less noise than absolute geographical location.

The device may also include other features in communication withcomputer 110, such as an accelerometer, gyroscope or anotherdirection/speed detection device 146 to determine the direction andspeed of the vehicle or changes thereto. By way of example only,acceleration device 146 may determine its pitch, yaw or roll (or changesthereto) relative to the direction of gravity or a plane perpendicularthereto. The device may also track increases or decreases in speed andthe direction of such changes. The device's provision of location andorientation data as set forth herein may be provided automatically tothe user, computer 110, other computers and combinations of theforegoing.

The computer 110 may control the direction and speed of the vehicle bycontrolling various components. By way of example, if the vehicle isoperating in a completely autonomous mode, computer 110 may cause thevehicle to accelerate (e.g., by increasing fuel or other energy providedto the engine), decelerate (e.g., by decreasing the fuel supplied to theengine or by applying brakes) and change direction (e.g., by turning thefront two wheels).

Computer 110 may also control status indicators, in order to convey thestatus of the vehicle and its components to a passenger of vehicle 101.For example, as shown in FIG. 2, vehicle 101 may be equipped with adisplay 225 for displaying information relating to the overall status ofthe vehicle, particular sensors, or computer 110 in particular. Thedisplay 225 may include computer generated images of the vehicle'ssurroundings including, for example, the status of the computer(cruise), the vehicle itself, roadways, intersections, as well as otherobjects and information.

Computer 110 may use visual or audible cues to indicate whether computer110 is obtaining valid data from the various sensors, whether thecomputer is partially or completely controlling the direction or speedof the car or both, whether there are any errors, etc. Vehicle 101 mayalso include a status indicating apparatus, such as status bar 230, toindicate the current status of vehicle 101. Status bar 230 displays “D”and “2 mph” indicating that the vehicle is presently in drive mode andis moving at 2 miles per hour. In that regard, the vehicle may displaytext on an electronic display, illuminate portions of vehicle 101, orprovide various other types of indications. In addition, the computermay also have external indicators which indicate whether, at the moment,a human or an automated system is in control of the vehicle, that arereadable by humans, other computers, or both.

Returning to FIG. 1, computer 110 may be an autonomous driving computingsystem capable of communicating with various components of the vehicle.For example, computer 110 may be in communication with the vehicle'sconventional central processor 126 and may send and receive informationfrom the various systems of vehicle 101, for example the braking 180,acceleration 182, signaling 184, and navigation 186 systems in order tocontrol the movement, speed, etc. of vehicle 101. In addition, whenengaged, computer 110 may control some or all of these functions ofvehicle 101 and thus be fully or partially autonomous. It will beunderstood that although various systems and computer 110 are shownwithin vehicle 101, these elements may be external to vehicle 101 orphysically separated by large distances.

As shown in FIG. 2, the interior of the autonomous vehicle may includeall of the features of a non-autonomous vehicle, for example: a steeringapparatus, such as steering wheel 210; a navigation display apparatus,such as navigation display 215; and a gear selector apparatus, such asgear shifter 220.

The vehicle may include components for detecting objects external to thevehicle such as other vehicles, obstacles in the roadway, trafficsignals, signs, trees, etc. The detection system may include lasers,sonar, radar, cameras or any other detection devices. For example, ifthe vehicle is a small passenger car, the car may include a lasermounted on the roof or other convenient location. In one aspect, thelaser may measure the distance between the vehicle and the objectsurfaces facing the vehicle by spinning on its axis and changing itspitch. The vehicle may also include various radar detection units, suchas those used for adaptive cruise control systems. The radar detectionunits may be located on the front and back of the car as well as oneither side of the front bumper. In another example, a variety ofcameras may be mounted on the car at distances from one another whichare known so that the parallax from the different images may be used tocompute the distance to various objects which are captured by 2 or morecameras. These sensors allow the vehicle to understand and potentiallyrespond to its environment in order to maximize safety for passengers aswell as objects or people in the environment.

Many of these sensors provide data that is processed by the computer inreal-time, that is, the sensors may continuously update their output toreflect the environment being sensed at or over a range of time, andcontinuously or as-demanded provide that updated output to the computerso that the computer can determine whether the vehicle's then-currentdirection or speed should be modified in response to the sensedenvironment.

FIG. 3 illustrates a particular embodiment for a small passenger vehicle101 that includes lasers 310 and 311, mounted on the front and top ofthe vehicle, respectively. Laser 310 may have a range of approximately150 meters, a thirty degree vertical field of view, and approximately athirty degree horizontal field of view. Laser 311 may have a range ofapproximately 50-80 meters, a thirty degree vertical field of view, anda 360 degree horizontal field of view. The lasers may provide thevehicle with range and intensity information which the computer may useto identify the location and distance of various objects. In one aspect,the lasers may measure the distance between the vehicle and the objectsurfaces facing the vehicle by spinning on its axis and changing itspitch.

The vehicle may also include various radar detection units, such asthose used for adaptive cruise control systems. The radar detectionunits may be located on the front and back of the car as well as oneither side of the front bumper. As shown in the example of FIG. 3,vehicle 101 includes radar detection units 320-323 located on the side(only one side being shown), front and rear of the vehicle. Each ofthese radar detection units may have a range of approximately 200 metersfor an approximately 18 degree field of view as well as a range ofapproximately 60 meters for an approximately 56 degree field of view.

In another example, a variety of cameras may be mounted on the vehicle.The cameras may be mounted at predetermined distances so that theparallax from the images of 2 or more cameras may be used to compute thedistance to various objects. As shown in FIG. 3, vehicle 101 may include2 cameras 330-331 mounted under a windshield 340 near the rear viewmirror (not shown). Camera 330 may include a range of approximately 200meters and an approximately 30 degree horizontal field of view, whilecamera 331 may include a range of approximately 100 meters and anapproximately 60 degree horizontal field of view.

Each sensor may be associated with a particular sensor field in whichthe sensor may be used to detect objects. FIG. 4A is a top-down view ofthe approximate sensor fields of the various sensors. FIG. 4B depictsthe approximate sensor fields 410 and 411 for lasers 310 and 311,respectively based on the fields of view for these sensors. For example,sensor field 410 includes an approximately 30 degree horizontal field ofview for approximately 150 meters, and sensor field 411 includes a 360degree horizontal field of view for approximately 80 meters.

FIG. 4D depicts the approximate sensor fields 420A-423B and for radardetection units 320-323, respectively, based on the fields of view forthese sensors. For example, radar detection unit 320 includes sensorfields 420A and 420B. Sensor field 420A includes an approximately 18degree horizontal field of view for approximately 200 meters, and sensorfield 420B includes an approximately 56 degree horizontal field of viewfor approximately 80 meters. Similarly, radar detection units 321-323include sensor fields 421A-423A and 421B-423B. Sensor fields 421A-423Ainclude an approximately 18 degree horizontal field of view forapproximately 200 meters, and sensor fields 421B-423B include anapproximately 56 degree horizontal field of view for approximately 80meters. Sensor fields 421A and 422A extend passed the edge of FIGS. 4Aand 4D.

FIG. 4C depicts the approximate sensor fields 430-431 cameras 330-331,respectively, based on the fields of view for these sensors. Forexample, sensor field 430 of camera 330 includes a field of view ofapproximately 30 degrees for approximately 200 meters, and sensor field431 of camera 430 includes a field of view of approximately 60 degreesfor approximately 100 meters.

In another example, an autonomous vehicle may include sonar devices,stereo cameras, a localization camera, a laser, and a radar detectionunit each with different fields of view. The sonar may have a horizontalfield of view of approximately 60 degrees for a maximum distance ofapproximately 6 meters. The stereo cameras may have an overlappingregion with a horizontal field of view of approximately 50 degrees, avertical field of view of approximately 10 degrees, and a maximumdistance of approximately 30 meters. The localization camera may have ahorizontal field of view of approximately 75 degrees, a vertical fieldof view of approximately 90 degrees and a maximum distance ofapproximately 10 meters. The laser may have a horizontal field of viewof approximately 360 degrees, a vertical field of view of approximately30 degrees, and a maximum distance of 100 meters. The radar may have ahorizontal field of view of 60 degrees for the near beam, 30 degrees forthe far beam, and a maximum distance of 200 meters.

As described below, autonomous vehicle's computer system is able totrack a detected object's movements and record data associated with theobject and its movements. Once a nearby object is detected, system 100may determine the object's type, for example, a traffic cone, person,car, truck or bicycle. Objects may be identified by using objectclassification data 116 (shown in FIG. 1), which may consider variouscharacteristics of the detected objects, such as the size of an object,the speed of the object (bicycles do not tend to go faster than 40 milesper hour or slower than 0.1 miles per hour), the heat coming from thebicycle (bicycles tend to have rider that emit heat from their bodies),etc. In addition, the object may be classified based on specificattributes of the object, such as information contained on a licenseplate, bumper sticker, or logos that appear on the vehicle.

In one example, autonomous driving system 100 is operable to predictanother vehicle's future movement based solely on the other vehicle'sinstant direction, acceleration/deceleration and velocity, e.g., thatthe other vehicle's current direction and movement will continue.However, autonomous vehicle may access a behavior model that providesthe probability of one or more actions being taken by the detectedobject. Specifically, system 100 may predict a detected vehicle's futuremovement by analyzing data relating to the other vehicle's currentsurroundings and determining how the other vehicle will likely respondto those surroundings. In other words, autonomous driving system 100uses an object-centric, view of the object's environment, in that thesystem determines what the other vehicles are perceiving in order tobetter predict how those vehicles will behave.

The system may also collect information that is independent of otherdetected object's in order to predict the an object's next action. Byway of example, if the vehicle determines that another object is abicycle that is beginning to ascend a steep hill in front of thevehicle, the computer may predict that the bicycle will soon slowdown—and will slow the vehicle down accordingly—regardless of whetherthe bicycle is currently traveling at a somewhat high speed.

The computer may cause the vehicle to take particular actions inresponse to the predicted actions of the surrounding objects. Forexample, if the computer 110 determines that the other car is turning atthe next intersection as noted above, the computer may slow the vehicledown as it approaches the intersection. In this regard, the predictedbehavior of other objects is based not only on the type of object andits current trajectory, but also based on some likelihood that theobject may obey traffic rules or pre-determined behaviors. In anotherexample, the system may include a library of rules about what objectswill do in various situations. For example, a car in a left-most lanethat has a left-turn arrow mounted on the light will very likely turnleft when the arrow turns green. The library may be built manually, orby the vehicle's observation of other vehicles (autonomous or not) onthe roadway. The library may begin as a human built set of rules whichmay be improved by the vehicle's observations. Similarly, the librarymay begin as rules learned from vehicle observation and have humansexamine the rules and improve them manually. This observation andlearning may be accomplished by, for example, tools and techniques ofmachine learning. The rules library may be included in computer 110 ormay alternatively be accessible to the vehicle 101 via a remote server,such as server 142 of FIG. 7.

In addition to processing data provided by the various sensors, thecomputer may rely on environmental data that was obtained at a previouspoint in time and is expected to persist regardless of the vehicle'spresence in the environment. As shown in FIG. 1, data 134 may includedetailed map information 114, such as highly detailed maps identifyingthe shape and elevation of roadways, lane lines, intersections,crosswalks, speed limits, traffic signals, buildings, signs, real timetraffic information, or other such objects and information. For example,the map information may include explicit speed limit informationassociated with various roadway segments. The speed limit data may beentered manually or scanned from previously taken images of a speedlimit sign using, for example, optical-character recognition. The mapinformation may include three-dimensional terrain maps incorporating oneor more of objects listed above. For example, the vehicle may determinethat another car is expected to turn based on real-time data (e.g.,using its sensors to determine the current GPS position of another car)and other data (e.g., comparing the GPS position with previously-storedlane-specific map data to determine whether the other car is within aturn lane).

The computer 110 may also access data 134 relating to certain types ofobjects that the vehicle 101 may encounter. As described above, thesensors of vehicle 101 may be used to identify, track and predict themovements of pedestrians, bicycles, vehicles, or other objects in oraround the roadway. These objects may have particular behavior patternsthat depend on the nature of the object. For example, a bicycle islikely to react differently than a tractor-trailer in a number of ways.Specifically, a bicycle is more likely to make erratic movements whencompared with a tractor-trailer. Accordingly, in predicting an objectsbehavior, computer 110 may access behavior model data 136 and objectdata 116 that contains numerous object classifications, such aspedestrians, bicycles, cars, tractor-trailers, etc. For eachclassification, the behavior model data 136 may also contain behaviorinformation that indicates how an object having a particularclassification is likely to behave in a given situation. Vehicle 101 maythen autonomously respond to the object based, in part, on the predictedbehavior.

In addition to classifying the object, vehicle 101 may track a currentstate of the object. The object's state may include information used todetermine the object's classification, but any type of environmental orcontextual information may be used to determine the object's currentstate, such as the object's speed, route the object has traveled, natureof the roadway on which the object is traveling, or the object's use ofheadlights or blinkers.

FIG. 5 illustrates an example of vehicle 101 detecting surroundingobjects. Using the sensors described above, autonomous vehicle 101 iscapable of detecting surrounding vehicles 510-550. As indicated bydirectional arrows A1, B1, C, and D, vehicle 510 and vehicle 520 arecurrently traveling along the center lane, while autonomous vehicle 101and vehicle 530 are traveling north in the right-hand lane. Upondetecting the surrounding vehicles, computer system 110 may track andrecord the position and movement (e.g. velocity, heading, andacceleration) of each surrounding vehicle.

The position and movement data for the detected vehicles 510-550 may bestored in database 116 of the autonomous driving computer system, asshown in FIG. 1. The autonomous vehicle may then use this data tomaneuver vehicle 101 in a way that avoids collisions with nearbyvehicles. In accordance with one embodiment, computer system 110 mayfilter the position and movement data, so as to identify a vehicle'sposition and movement during particular actions of interest.

For example, database 138 may include a set of actions or behaviors ofinterest, such as the vehicle changing lanes or routes, and instructions132 may allow for computer system 110 to identify when a detectedvehicle has performed one or more of the behaviors of interest. Inparticular, computer system 110 of FIG. 1 may access the recordedposition and movement stored in database 116, as well as a road graph ofthe environment stored in database 114. By combining both sets of data,computer system 110, may then determine when one or more of the keybehaviors have occurred.

Returning to FIG. 5, vehicle 101 may record the position and movementdata of vehicle 510 and vehicle 520. In doing so, vehicle 101 can detectthat vehicle 520 has de-accelerated and is changing its heading in thedirection of arrow A2. Similarly, vehicle 101 detects that vehicle 510has changed its heading as provided by arrow B2. Vehicle 101 may thenaccess stored road graph data that represents vehicle the currentenvironment of vehicles 510 and 520, such as road graph 600 illustratedin FIG. 6. Based on the position and movement of vehicles 510 and 520,relative to vehicle 101, the computer system of vehicle 101 mayassociate vehicles 510 and 520 with the center lane of City Highway. Thecomputer system of vehicle 101 may then compare the changed heading ofvehicle 520 with the stored road graph to determine that vehicle 520 isturning from City Highway to Through Street, at point 610. Likewise, thecomputer system compares the change in heading of vehicle 510 with map600 to determine that the vehicle has changed from the center lane tothe adjacent lane to the right, at point 620.

In this way, vehicle 101 may associate and track all surroundingvehicles with a particular road graph element, such as a lane of travelor intersection. For example, dashed line 630 in FIG. 6 represents thepath of travel of vehicle 510. Provided that vehicle 510 remains withinan area that can be detected by vehicle 101, this path may be recordedby vehicle 101's computer system. By combining the vehicle's movementdata with map 600, vehicle 101 identifies instances along vehicle 510'spath where it has performed one or more behaviors of interest. Forexample, the computer system of vehicle 101 may identify any change inroad element as a behavior of interest, such as vehicle 510 travelingfrom City Highway, to First Avenue, and then to River Road. In addition,vehicle 101 may identify when vehicle 510 has changed lanes on aparticular roadway, such as at points 620 and 650.

Vehicle 101 may also filter the data collected for vehicle 510 so thatit only contains instances where vehicle 510 has performed an action ofinterest. As provided by line 630 on map 600, vehicle 510 changes it'sheading around point 640, as it begins to travel from a north-westdirection to a more south-west direction. While vehicle 101 will collectdata regarding vehicle 510's change in heading, computer 110 will alsodetermine that the change in heading does not correspond to an action ofinterest, as vehicle 510 merely travels along the same road graphelement. Vehicle 101 may, in turn, exclude the data corresponding tovehicle 510's change in heading at point 640 as being recorded as anaction of interest.

In another embodiment, autonomous vehicle 101 may transport itself,passengers, and/or cargo between two locations by following a route. Forexample, a driver may input a destination and activate an autonomousmode of the vehicle. In response, the vehicle's computer 110 maycalculate a route using a map, its current location, and thedestination. Based on the route (or as part of the route generation),the vehicle may determine a control strategy for controlling the vehiclealong the route to the destination. In accordance with one embodiment,computer system 110 may control the autonomous vehicle 101 to takeparticular actions in response to the actions of the surrounding objectsthat have been identified as performing a behavior of interest. Forexample, by changing lanes as provided by arrow B2 of FIG. 5, vehicle101 may identify that vehicle 510 has performed a behavior of interestand thereby reevaluate the current control strategy of vehicle 101.Specifically, computer system 110 may determine that vehicle 101 shoulddecelerate in order to avoid travelling in vehicle 510's blind-spot,given that vehicle 510 is now travelling in the adjacent lane.

As another example, vehicle 520 may come to a stop for a period of timebefore making the left-hand turn designated by arrow A2. Computer system110 may identify this action as a behavior of interest, depending onwhich road element vehicle 520 is travelling. Specifically, if vehicle520 is determined to be in a left-hand turn lane, vehicle 101 may notidentify vehicle 520 having stopped will not as a behavior of interest.However, if vehicle 520 was travelling one lane over to the right, thefact that it has stopped could indicate that there is a backup ahead.Accordingly, vehicle 101 may record instances where the detected objectshave performed actions of interest.

In addition to determining instance when a detected object has performedan action of interest, autonomous vehicle 101 may also record how eachnearby vehicle perceives its surroundings. In the case of vehicle 510,nearby objects include vehicles 510, 520, 530, 540, and 550, as well asautonomous vehicle 101. Accordingly, autonomous vehicle 101 uses thesensors described above to collect position and movement data for eachof these nearby objects relative to vehicle 510. The relative positionand movement data is object-centric, in that it represents the detectedobject's “world-view”, based on how the object views its surroundings.This world-view data may be stored in database 116, shown in FIG. 1.

The autonomous driving computer system 100 may also collect object datato create or further refine already existing behavior models used topredict the likely movements of detected objects, such as vehicles orpedestrians. For example, the collected object data may supplementinformation already used to initially construct the behavior model. Inother words, the supplemental object data may provide additionalstatistical information about the behavior of a detected object tofurther refine the probable behaviors of the object in the givenenvironment. Thus, as the autonomous vehicle 101 moves about itsenvironment, the supplemental state information may improve thereliability and predictability of a given behavior model.

The autonomous vehicle 101 may collect state information about detectedobjects regardless of whether the autonomous vehicle 101 is operating inan autonomous mode or a non-autonomous mode. Thus, whether theautonomous vehicle 101 is operating by itself or has a driver, theautonomous vehicle 101 may collect state and object information toformulate the aforementioned behavior models.

It should also be understood that object data may be collected by othermeans other than the autonomous vehicle 101. For example, object datamay be collected by a non-autonomous vehicle, satellite imagery, trafficreports, police reports, user-provided information, or other such means.In addition, these other data collection means by work in concert withthe autonomous vehicle 101 and may communicate the collected object datato the autonomous vehicle 101. For example, one or more sensors placedin a driving environment may communicate collected object data to theautonomous vehicle 101 when the autonomous vehicle 101 is in proximityto a given sensor, such as when the autonomous vehicle 101 passes asensor (e.g., where the sensor is placed in or underneath the roadway orwhere the sensor is placed on the side of the roadway.) Moreover, acombination of aforementioned or similar data collection means may beused in developing the disclosed behavior models.

The autonomous vehicle 101 may track and record object data in anynumber of driving environments, such as an elementary school crossing, ahighway interchange, a suburban intersection, a four-way stop signintersection, or any other driving environment. After tracking nearbyvehicles and/or pedestrians in one or more of these drivingenvironments, the autonomous vehicle 101 may communicate the collectedobject data to an object behavior model server 142, shown in FIG. 1.While the object behavior model server 142 is shown separately from theautonomous vehicle 101, it should be understood that the object behaviormodel server 142 may also be incorporated into the autonomous vehicle101, such as by being incorporated into the autonomous driving computersystem 110. Moreover, portions of the object behavior model server 142may also reside in the memory 130 of the autonomous driving computersystem 110. Combinations of the foregoing are also possible.

The behavior model server 142 is operative to develop behavior modelsfor the various classifications of objects based on the object data 116collected by the autonomous vehicle 101. As used in this disclosure, a“behavior model” may refer to the expected behavior of a givenclassification of objects in a particular driving environment. Forexample, the object behavior model server 142 may define a behaviormodel for passenger vehicles entering a four-way intersection. In thisexample, the behavior model may define the probability that a passengervehicle proceeds straight through the intersection, the probability thata passenger vehicle turns left at the intersection, and the probabilitythat the passenger vehicle turns right at the intersection.

The behavior model may further define the behavior of the passengervehicle, such as the approximate amount of time the passenger vehicle isexpected to stop at the intersection, the approximate amount of time thepassenger vehicle is expected to move through the intersection, theapproximate speed of the passenger as it turns or proceeds through theintersection, or other such behavior information. In this manner, abehavior model may provide the autonomous vehicle 101 with behaviorinformation such that the autonomous vehicle 101 may predict thebehavior of the corresponding classification of vehicle in a givendriving environment. Based on the behavior information provided by thebehavior model, the autonomous vehicle 101 may respond with an action ortake an action depending on the predicted behavior defined by thebehavior model.

In addition to how an object itself may behave, a behavior model mayprovide information as to how the autonomous vehicle 101 may react orbehave. For example, the behavior model may provide information thatinfluences whether the autonomous vehicle 101 accelerates, decelerates,turns right, turns left, enables an indicator (e.g., turning on aheadlight or turning light), disables an indicator (e.g., turning off aheadlight or turning light), or engages some other vehicle behavior(e.g., turning on/off windshield wipers, turning on/off fog lights,turning on/off high beams, etc.) Thus, a behavior model may not onlyincorporate information about what the behavior of the associated objectmay be, but the behavior model may also provide information that mayinfluence how the autonomous vehicle 101 vehicle may react.

In addition, behavior models for differing classifications of objectsmay provide information that leads to differing actions by theautonomous vehicle 101 depending on the type of object associated withthe behavior model. For example, a behavior model associated with apassenger vehicle classification may provide information that influencesthe autonomous vehicle 101 to take actions that are different for abehavior model associated with a motorcycle classification. Moreover,the actions may differ even when the predicted action of the object isthe same. For example, a motorcycle behavior model and a passengervehicle behavior model may define that the associated objects (i.e., themotorcycle and the passenger vehicle) may turn left under a givencircumstance, but that the behavior of the autonomous vehicle 101 may bedifferent for each circumstance. Thus, a behavior model may be furtherdefined by the type of object classification (e.g., motorcycle,passenger vehicle, commercial truck, etc.) assigned to the behaviormodel.

A behavior model may also generalize, or be specific to, a given drivingenvironment. For example, a behavior model may define the behavior of agiven object classification for a four-way intersection. However, thebehavior model may be more specific in that the behavior model maycorrespond to a particular geographic location having a four-wayintersection (e.g., the intersection of 5th Avenue and 42nd Street inNew York City, N.Y.). Thus, depending on the granularity and specificityof the behavior model, the behavior model may provide behaviorinformation regarding an object classification for a given type ofdriving environment, or may even provide behavior information regardingan object classification for a specific geographic location of a giventype of driving environment.

Moreover, a behavior model may be tailored or based on specific types ofdriving conditions. For example, a behavior model may provide behaviorinformation based on weather conditions (e.g., rain, snow, sleet, hail,fog, etc.) A behavior model based on a first weather condition (e.g.,rain) may provide information that differs from a behavior model basedon a second weather condition (e.g., snow). The differing informationmay be associated with the corresponding object classification of thebehavior model, the behavior of the autonomous vehicle 101, or acombination thereof. Other examples of driving conditions on which abehavior model may be based include the time of day, traffic conditions,whether there a blind corner is involved in a road/intersection, whethera hill is involved in a roadway, or other such driving conditions.

The object behavior model server 142 is operative to develop behaviormodels that do not yet exist for corresponding object classificationsand to refine already existing behavior models. With regard todeveloping new behavior models, the object behavior model server 142 mayinitially receive object data 116 (e.g., behavior and/or state data)from the autonomous vehicle 101. The object behavior model server 142may then determine whether a behavior model already exists for theobject classification corresponding to the received object data 116.Should the object behavior model server 142 not have an existingbehavior model, the object behavior model server 142 may then develop anew behavior model for the corresponding object classification. However,should a behavior model already exist for the corresponding objectclassification, the object behavior model server 142 may then refine thebehavior model by supplementing the behavior information in thealready-existing behavior model with the newly-received object data fromthe autonomous vehicle 101. In this manner, the behavior models residingon the object behavior model server 142 may be continuously refined toimprove the accuracy of the developed behavior models and the behaviorprediction of the autonomous vehicle 101.

FIG. 7 illustrates one example of the object behavior model server 142.In one embodiment, the object behavior model server 142 may include aprocessor 702 and a memory 704. The object behavior model server 142 mayalso include other components typically present in a general purposecomputer. The memory 704 may store information accessible by theprocessor 702, such as instructions 706 and data 708 that may beexecuted or otherwise used by the processor 702. The memory 704 may beof any type of memory operative to store information accessible by theprocessor 702, including a computer-readable medium, or other mediumthat stores data that may be read with the aid of an electronic device.Examples of the memory 704 include, but are not limited, a hard-drive, amemory card, ROM, RAM, DVD, or other optical disks, as well as otherwrite-capable and read-only memories. Systems and methods may includedifferent combinations of the foregoing, whereby different portions ofthe instructions and data are stored on different types of media.

The instructions 706 may be any set of instructions that may be executeddirectly (such as machine code) or indirectly (such as scripts) by theprocessor 702. For example, the instructions 706 may be stored ascomputer code on the computer-readable medium. In that regard, the terms“instructions” and “programs” may be used interchangeably herein. Theinstructions 706 may be stored in object code format for directprocessing by the processor 702, or in any other computer languageincluding scripts or collections of independent source code modules thatare interpreted on demand or compiled in advance. Functions, methods androutines of the instructions 706 are explained in more detail below.

The data 708 may be retrieved, stored, or modified by the processor 702in accordance with the instructions 706. For instance, although thedisclosed embodiments not limited by any particular data structure, thedata 708 may be stored in computer registers, in a relational databaseas a table having a plurality of different fields and records, XMLdocuments, flat files, or in any computer-readable format. By furtherway of example only, image data may be stored as bitmaps comprised ofgrids of pixels that are stored in accordance with formats that arecompressed or uncompressed, lossless (e.g., BMP) or lossy (e.g., JPEG),and bitmap or vector-based (e.g., SVG), as well as computer instructionsfor drawing graphics. The data 708 may comprise any informationsufficient to identify the relevant information, such as numbers,descriptive text, proprietary codes, references to data stored in otherareas of the same memory or different memories (including other networklocations) or information that is used by a function to calculate therelevant data.

The processor 702 may be any conventional processor that is commerciallyavailable. Alternatively, the processor 702 may be a dedicated devicesuch as an ASIC or FPGA. Although FIG. 7 functionally illustrates theprocessor 702, the memory 704, and other elements of the object behaviormodel server 142 as being within the same block, it will be understoodby those of ordinary skill in the art that the processor 702 and thememory 704 may actually comprise multiple processors and memories thatmay or may not be stored within the same physical housing. For example,the memory 704 may be a hard drive or other storage media located in ahousing different from that of the object behavior model server 142.

As discussed previously, the object behavior model server 142 may beoperative to develop new behavior models or refine already existingbehavior models based on the information collected by the autonomousvehicle 101. In one embodiment, data 708 stored in the memory 704includes received object data 710, one or more behavior models 712, anda detailed map 714. The received object data 710 may include the objectdata 116 collected by the autonomous vehicle 101, includingclassification of the object, actions of interest that have beenperformed by the object, and world-view data. In addition, the objectdata 710 may include object data collected by other means, such assatellite imagery, traffic reports, police reports, and other suchmeans.

The object behavior model server 142 may develop one or more of thebehavior models 712 based on the object data 710. As discussedpreviously, each classification of object (e.g., passenger vehicle,light truck, pedestrian, bicycle, etc.) may be associated with one ormore behavior models. In one embodiment, an object classification may beassociated with a number of behavior models, where each behavior modelcorresponds to a driving environment, including the object's currentworld-view and potential actions of interest that have recently beenperformed and are currently available to the object. For example, withreference to FIG. 5, a “passenger vehicle” object classification may beassociated with a behavior model for vehicle 510. The environment forvehicle 510 is the center lane of multiple-lane road 500, which wouldallow for various potential actions of interest, including vehicle 510changing into the adjacent right-hand lane.

In one embodiment of the object behavior model server 142, the objectbehavior model 142 may develop behavior models based on the expectedprobability that an object will perform a given action. For example,with reference to FIG. 5, the object behavior model server 142 mayreceive object data about one or more vehicles 510-550, such as thenumber times a vehicle of the same classification as vehicle 520 turned(e.g., followed path A2) or the number of times a vehicle of theclassification of vehicle 530 changed lanes (e.g., followed path B2). Inthis embodiment, the object behavior model server 142 may determine theexpected probability that a given classification of vehicle will turn bydetermining number of times the path A2 and dividing this number by thetotal number of paths taken. Similarly, the object behavior model server142 may determine the expected probability that a given classificationof vehicle will continue straight by determining the number of timespath B2 was taken and dividing this number by the total number of pathstaken.

However, a more robust behavior model may alternatively be implementedby server 142. The object data provided by the autonomous vehicleincludes the world-view data representing how each detected objectperceives its surroundings. This data may be used to further limit oralter the probable actions of each detected object. Specifically, server142 may use the world-view data, which provides an object-centricperspective of detected vehicle's environment, to determine how othervehicles or pedestrians might react, given their perception of othernearby objects.

For example, the autonomous vehicle 101 in FIG. 5 will provide server142 with object data for vehicle 510 that identifies nearby objects thatcan be perceived by vehicle 510, including vehicles 520, 530, 540, and550, as well as autonomous vehicle 101. The object data 710 will alsoinclude position and movement data for each of these nearby objectsrelative to vehicle 510. For example, server 142 may receive object data710 indicating that vehicle 520 has turned on its left-hand blinker oris slowing down as it approaches the next intersection. Server 142 may,in turn, predict that vehicle 520 intends to make a left-hand turn, asindicated by directional arrow A2. Based on this additional dataregarding vehicle 520, and by accessing the behavior model database 712,server 142 is capable of altering its prediction of vehicle 510's futuremovement. In particular, server 142 determines how often a passengervehicle would change lanes if placed in vehicle 510's position behindthe turning vehicle 520. Given that none of the other nearby vehiclesoccupy the center-lane in vehicle 510's direction of travel, thebehavior model database 712 may predict that in this situation vehicle510 has a high probability of changing into the center lane, asindicated by directional arrow B2, in order to avoid being impeded byvehicle 520.

Server 142 may perform a similar, object-centric behavior prediction foreach of the other nearby vehicles in FIG. 5. For example, server 142 maydetermine how a passenger vehicle would react if placed in the positionof vehicle 520, and determine that given the presence of vehicles 540and 550, vehicle 520 will need to come to a complete stop before makingthe left-hand turn along path A2. In turn, the behavior model 712 willlikely increase the probability of vehicle 510 changing lanes along pathB2, as it will be directly behind a stopped vehicle. In this way, server142 is able to determine how each nearby vehicle perceives itssurroundings and to adjust the predicted behavior of each vehicle basedon the predicted behavior of other nearby vehicles.

Server 142 may further alter the predicted behavior of vehicles 510 and520 based on previous actions of interest that have occurred. Forexample, if vehicle 510 had very recently changed from the adjacentright-hand lane into its current lane, the behavior model mightdetermine the probability of a vehicle 510 immediately changing backinto the adjacent right-hand lane, along path B2. If this is a lowprobability event, the behavior model might lower the probability ofvehicle 510 changing lanes. The behavior model may also rely oninformation that is independent of other detected object's in order topredict the an object's next action. By way of example, if the vehicledetermines that another object is a bicycle that is beginning to ascenda steep hill in front of the vehicle, the computer may predict that thebicycle will soon slow down—and will slow the vehicle downaccordingly—regardless of whether the bicycle is currently traveling ata somewhat high speed.

As is understood in the art, determining expected probabilities that aparticular path will be followed is one statistical technique that maybe used to predict the behavior of a given object or to provideinformation for influencing the behavior of the autonomous vehicle 101.In addition, a variety of statistical estimation techniques may be used,alone or in combination, including any existing statistical estimation,machine learning, classification or other optimization methods. Furtherexamples include regression analysis, neural networks, decision trees,boosting, support vector machines, k-nearest neighbors, combinations ofthe foregoing, and other such examples.

Server 142 may use separate behavior models for differentclassifications of objects, including passenger vehicles,tractor-trailers, bicycles, or pedestrians. For example, a pedestrian orcyclist will not react to its surroundings in the same way as autonomousvehicle 101. In addition, an autonomously navigated station wagon oftenwill not react the same was as a sports car that is being controlled bya driver. Accordingly, database 712 may include different object-centricbehavior models for different classifications of objects, includingautonomous or non-autonomous vehicles.

FIG. 8 illustrates one example of logic flow 800 for developing behaviormodels for the various object classifications. Initially, the objectbehavior model server 142 may receive object data—including positiondata, actions-of-interest data, and world-view data—from the autonomousvehicle 101 (Block 802). Thereafter, the object behavior model server142 may analyze the received object data to determine the objectclassifications identified by the object data (Block 804). The objectbehavior model server 142 may then identify the driving environments ofthe identified object classifications based on the position data and theworld-view data (Block 806). The object behavior model server 142 thenidentifies a behavior of interest corresponding to an action taken bythe object based on the object data and on map information identifying aroad element corresponding to the location of the object (Block 807) andidentifying the behavior of interest is further based on a comparisonbetween the road element and the movement information with thecomparison indicating that the object has performed the behavior ofinterest (Block 809).

Using the identified object classifications and identified drivingenvironments, the object behavior model server 142 may determine whethercorresponding behavior models already exist (Block 808). Shouldcorresponding behavior models already exist, (Yes at Block 808) theobject behavior model server 142 may refine the already existingbehavior models using the received object data as supplemental data.Refining an already existing behavior model may include re-determiningthe behavior model for a given classification and/or driving environmentusing the already existing behavior model and the received object dataas supplemental data (Block 810). In other words, the already existingbehavior model may be updated with the object's world-view,classification, and actions of interest data. However, should a behaviormodel not exist for the corresponding object classification and/ordriving environment (No at Block 808) the object behavior model server142 may then develop a corresponding behavior model using the receivedobject data (Block 812). The object behavior model server 142 may thencommunicate the updated or newly created behavior model to theautonomous vehicle 101 for use in predicting the behavior of detectedobjects (Block 814). The object behavior model server 142 may thenmaneuvering the vehicle based on the behavior model and the identifiedbehavior of interest (Block 816).

In this manner, the monitoring and recording of detected object behaviorby the autonomous vehicle 101 may facilitate the development of robustbehavior models for predicting the behavior of the monitored andrecorded objects based on how the object perceives its surroundings, aswell as based on the previous actions of interest that it has performed.Moreover, the behavior models may be tailored to specific drivingenvironments along various parameters. Because the behavior models maybe customized with parameters specific to a given driving environment,the autonomous vehicle 101 may better accurately predict the behavior ofa detected object and take actions (e.g., braking, accelerating, etc.)in response to the behavior of the detected object. In addition, sincethe autonomous vehicle 101 may continuously collect behavior informationabout detected objects, already existing behavior models may be furtherrefined to better reflect the behavior of real-world objects. Thus, thepartnership between the behavior model behavior server 142 and theautonomous vehicle 101 forms a symbiotic relationship in which themonitoring by the autonomous vehicle 101 enhances the behavior modelsdetermined by the object behavior model server 142, and the updatedbehavior models determined by the object behavior model server 142improve the performance and reaction time of the autonomous vehicle 142.

Autonomous vehicle 101 may transport itself, passengers, and/or cargobetween two locations by following a route. For example, a driver mayinput a destination and activate an autonomous mode of the vehicle. Inresponse, the vehicle's computer 110 may calculate a route using a map,its current location, and the destination. Based on the route (or aspart of the route generation), the vehicle may determine a controlstrategy for controlling the vehicle along the route to the destination.For example, the control strategy may include where to turn, at whatspeeds to travel, what lane to travel in, where to look for trafficsignals, where to stop for intersections or stop signs, etc. Flowdiagram 900 of FIG. 9 provides an example by which vehicle 101 may beautonomously controlled in response to the model behavior data. Asprovided in Block 910, vehicle 101 implements the determined controlstrategy by traveling along a determined route. This control strategymay be based on a behavior model that is currently residing in thememory of autonomous vehicle 101. While traveling in accordance with thecontrol strategy, vehicle 101 detects the presence of numerous objectswithin one or more of the vehicle's sensor fields (Block 915). Upondetecting the objects, the computer 110 may classify the object based onthe data received by vehicle 101's sensors and determine environmentalinformation, such as its position and movement relative to other objects(Block 920). For example, the sensor data could be used to classifyobjects as being a pedestrian, bicycle, or vehicle. As described above,the vehicle's computer 110 also uses the sensor data to determine theobject's current state, such as speed, heading, and acceleration. Upondetermining the objects classification and current state, the computer110 may associate and track the detected objects with a road graphelement by accessing a map in database 114 (FIG. 1), and determiningwhether the detected objects have performed a behavior of interest bycomparing the object's position and movement with the map's road graphelement (Block 925). Autonomous vehicle 101 records object data for eachof the detected objects, including actions of interest and world-viewdata relating to how the objects perceive their surroundings (Block930).

Autonomous vehicle may then transmit the recorded object data to aremote behavior model server (Block 935). Autonomous vehicle may thendetermine whether the server has provided a modified behavior model(Block 740). If the server has provided a modified behavior model,computer 110 may then alter the control strategy of autonomous vehicle101 (Block 745). If not, Blocks 715 through 745 may then be repeateduntil autonomous vehicle 101 has reached its destination or theautonomous control has be otherwise terminated (Block 750). In this way,vehicle 101 may further alter the control strategy upon any of thedetected vehicles performing an action of interest.

As shown in FIG. 2, Vehicle 101 may include one or more user inputdevices that enable a user to provide information to the autonomousdriving computer 110. For example, a user, such as passenger 290, mayinput a destination (e.g., 123 Oak Street) into the navigation systemusing touch screen 217 or button inputs 219. In another example, a usermay input a destination by identifying the destination. In that regard,the computer system may extract the destination from a user's spokencommand (e.g., by stating or inputting “De young museum”).

The vehicle may also have various user input devices for activating ordeactivating one or more autonomous driving modes. In some examples, thedriver may take control of the vehicle from the computer system byturning the steering wheel, pressing the acceleration or decelerationpedals. The vehicle may further include a large emergency button thatdiscontinues all or nearly all of the computer's decision-making controlrelating to the car's velocity or direction. In another example, thevehicle's shift knob may be used to activate, adjust, or deactivatethese autonomous modes.

In one aspect, the features described above may be used in combinationwith larger vehicles such as trucks, tractor trailers, or passengerbusses. For such vehicles, the system may consider additionalinformation when computing how to control the vehicle safely. Forexample, the physical attributes of a tractor trailer, such as itsarticulation and changing weight, may cause it to maneuver verydifferently than smaller passenger cars. Larger vehicles may requirewider turns or different levels of acceleration and braking in order toavoid collisions and maneuver safely. The computer may consider thegeometry of the vehicle when calculating and executing maneuvers such aslane changes or evasive actions.

The vehicle may be only partially autonomous. For example, the drivermay select to control one or more of the following: steering,acceleration, braking, and emergency braking.

The vehicle may also address driver impairment. For example, if a driverhas been unresponsive, has reduced cognitive abilities, or has beendetected as having fallen asleep, the vehicle may attempt to wake orotherwise prompt the driver to respond. By way of example only, a cameracapturing the driver's face may be used to determine whether thedriver's eyes have remained closed for an extended period of time. Ifthe driver remains unresponsive, the computer may cause the vehicleslow, stop or pull over to a safe location, or may assume control overthe vehicle's direction or speed to avoid a collision.

In another example, the system may be always on with the driverprimarily in control of the vehicle, but only intervene and take actionwhen the vehicle detects that there is an emergency situation. Forexample, if a thick fog reduces the visibility of the driver, thevehicle may use its sensors to detect the presence of objects in thevehicle's path. If the vehicle determines that a collision is imminentyet the driver has taken no action to avoid the collision, the vehiclemay provide the driver with a warning and, absent further correction bythe driver, take one or more evasive actions such as slowing, stoppingor turning the vehicle.

The vehicle may also improve driver performance while the vehicle isunder the control of a driver. For example, if a driver fails tomaintain the vehicle's position within a lane without using a turnsignal, the vehicle may slightly adjust the steering in order to smoothout the vehicle's movements and maintain the vehicle's position withinthe lane. Thus, the vehicle may mitigate the likelihood of undesirableswerving and erratic driving. In another embodiment, if a driver is on acourse to change lanes but has not activated the turn signal, thevehicle may automatically activate the turn signal based on the detectedcourse of the vehicle.

The vehicle may also be used to train and test drivers. For example, thevehicle may be used to instruct new drivers in the real world, but maycontrol various aspects of the ride in order to protect the new driver,people external to the car, and external objects. In another example,new drivers may be required to control the vehicle safely for someperiod. The vehicle may then determine whether the user is a safe driverand thus determine whether the user is ready to be a licensed driver.

The vehicle may also park itself. For example, the map information mayinclude data describing the location of parking spots along a roadway orin a parking lot. The computer may also be configured to use its sensorsto determine potential parking spots, such as causing the vehicle totravel down a road and checking for painted lines along a street thatindicate an open parking space. If computer determines another vehicleor object is not within the spot, the computer may maneuver the vehicleinto the parking spot by controlling the steering and speed of thevehicle. Using the method described above, the vehicle may also classifyany objects that are near the potential parking spot, and position thevehicle within the parking spot based on those surrounding objects. Forexample, the vehicle may position itself closer to an adjacent bicyclethan it would an adjacent truck.

The vehicle may also have one or more user interfaces that allow thedriver to reflect the driver's driving a style. For example, the vehiclemay include a dial which controls the level of risk or aggressivenesswith which a driver would like the computer to use when controlling thevehicle. For example, a more aggressive driver may want to change lanesmore often to pass cars, drive in the left lane on a highway, maneuverthe vehicle closer to the surrounding vehicles, and drive faster thanless aggressive drivers. A less aggressive driver may prefer for thevehicle to take more conservative actions, such as somewhat at or belowthe speed limit, avoiding congested highways, or avoiding populatedareas in order to increase the level of safety. By manipulating thedial, the thresholds used by the computer to calculate whether to passanother car, drive closer to other vehicles, increase speed and the likemay change. In other words, changing the dial may affect a number ofdifferent settings used by the computer during its decision makingprocesses. A driver may also be permitted, via the user interface 225,to change individual settings that relate to the driver's preferences.In one embodiment, insurance rates for the driver or vehicle may bebased on the style of the driving selected by the driver.

Aggressiveness settings may also be modified to reflect the type ofvehicle and its passengers and cargo. For example, if an autonomoustruck is transporting dangerous cargo (e.g., chemicals or flammableliquids), its aggressiveness settings may be less aggressive than a carcarrying a single driver—even if the aggressive dials of both such atruck and car are set to “high.” Moreover, trucks traveling across longdistances over narrow, unpaved, rugged or icy terrain or vehicles may beplaced in a more conservative mode in order reduce the likelihood of acollision or other incident.

In another example, the vehicle may include sport and non-sport modeswhich the user may select or deselect in order to change theaggressiveness of the ride. By way of example, while in “sport mode”,the vehicle may navigate through turns at the maximum speed that issafe, whereas in “non-sport mode”, the vehicle may navigate throughturns at the maximum speed which results in g-forces that are relativelyimperceptible by the passengers in the car.

The vehicle's characteristics may also be adjusted based on whether thedriver or the computer is in control of the vehicle. For example, when aperson is driving manually the suspension may be made fairly stiff sothat the person may “feel” the road and thus drive more responsively orcomfortably, while, when the computer is driving, the suspension may bemade such softer so as to save energy and make for a more comfortableride for passengers.

The vehicle may include a sleeping mode that allows the driver to givefull control of the vehicle to the computer so that the driver may sleepor reduce his or her focus on the roadway. For example, the vehicle mayinclude a user input device that allows the user to input informationsuch as the duration of the sleep mode, e.g., 20 minutes, 4 hours, 8hours, etc. In response, the vehicle may drive slower or on lesstraveled roadways, select a route that will get the driver to thedestination in the identified period, or select a route which that avoidbumps or other disturbances to the driver.

The driver may also select to have his or her vehicle communicate withother devices. As shown in FIG. 10, vehicle 101 may communicate over anetwork 960 with devices such as a remote server 142, a personalcomputer 965, a mobile device 970, or another autonomous vehicle 102. Inaddition, vehicles, such as vehicle 101 and vehicle 102, may wirelesslytransmit information directly to nearby vehicles using radio, cellular,optical or other wireless signals. Alternatively, vehicles maycommunicate with each via nodes that are shared among multiple vehicles,e.g., by using cell towers to call other cars or transmit and sendinformation to other cars via the Internet. The transmitted informationbetween vehicles may include, for example, data describing the vehicleor the vehicle's environment. In addition to receiving updated behaviormodel data from server 142, vehicle 101 may also receive updated map orobject data via network 960. Computer system 110, of FIG. 1, may then beupdated, and the new data may be used in controlling the vehicleautonomously, such as through implementation of flow diagram 900.

In one example, a driver of a first vehicle may select an option toallow other vehicles on the roadway to transmit information from thevehicle's sensors or computer. This information may include detailsabout the first vehicle's environment such as detected objects, trafficconditions, or construction. The information transmitted to othervehicles may be sensor data unprocessed by the first computer orinformation previously processed by the first computer in order toreduce the time needed to obtain and process the information at a secondvehicle. If the second autonomous vehicle is behind the first vehicle,it may use the information to determine how to maneuver the vehicle. Byway of example, if the first vehicle is only a few car lengths in frontof the second vehicle and it detects a moving object, the first vehiclemay transmit information relating to the moving object to the secondvehicle. If the second vehicle determines that the object is movingtowards the second vehicle's path, the second vehicle may slow down. Yetfurther, if the second vehicle is a few miles behind the first vehicleand the first vehicle determines that it is in a traffic jam (e.g., bydetermining that its speed is substantially less than the road's speedlimit), the second vehicle may select an alternate route.

In addition to cooperatively driving together in lines, autonomousvehicles may also communicate in order to increase convenience andsafety on the roadways. For example, autonomous vehicles may be able todouble (two vehicles in a row) and triple park (three vehicles in a row)next to other autonomous vehicles. When a driver would like to use avehicle which is parked in or surrounded by other autonomous vehicles,the driver's vehicle may send a signal instruction the other vehicles tomove out of the way. The vehicles may respond by cooperativelymaneuvering to another location in order to allow the driver's vehicleto exit and may return to park again.

In another example, the cooperation mode may be used to promote smarterroad crossings. For example, if several autonomous vehicles areapproaching and intersection, the right-of-way problem, or which vehicleshould be next to enter the intersection, may be calculated anddetermined cooperatively among the several vehicles. In another example,traffic signals may change quickly, such as within only a few seconds orless, to allow more vehicles to pass through an intersection in multipledirections. The vehicle may only need the traffic signal to be green fora second or less in order to pass through the intersection at highspeeds.

As these number and usage of these autonomous vehicles increases,various sensors and features may be incorporated into the environment toincrease the perception of the vehicle. For example, low-cost beacontransmitters may be placed on road signs, traffic signals, roads orother highway infrastructure components in order to improve thecomputer's ability to recognize these objects, their meaning, and state.Similarly, these features may also be used to provide additionalinformation to the vehicle and driver such as, whether the driver isapproaching a school or construction zone. In another example, magnets,RFID tags or other such items may be placed in the roadway to delineatethe location of lanes, to identify the ground speed vehicle, or increasethe accuracy of the computer's location determination of the vehicle.

Autonomous vehicles may also be controlled remotely. For example, if thedriver is asleep, the sensor data may be sent to a third party so thatvehicle may continue to have a responsive operator. While delay andlatency may make this type of telemetry driving difficult, it may forexample be used in emergency situations or where the vehicle has gottenitself stuck. The vehicle may send data and images to a central officeand allow a third party to remotely drive the vehicle for a short perioduntil the emergency has passed or the vehicle is no longer stuck.

As these and other variations and combinations of the features discussedabove can be utilized without departing from any method, system, orapparatus set forth in the claims, the foregoing description ofexemplary embodiments should be taken by way of illustration rather thanby way of limitation of the claims. It will also be understood that theprovision of examples (as well as clauses phrased as “such as,” “e.g.”,“including” and the like) should not be interpreted as limiting thescope of the claims to the specific examples; rather, the examples areintended to illustrate only some of many possible aspects.

The invention claimed is:
 1. A method of maneuvering a vehicle, themethod comprising: receiving, by one or more processors, object data,wherein the object data comprises a location of an object in anenvironment of the vehicle and movement information for the object;determining, from the object data, a classification of the object, theclassification being a type of a vehicle or a pedestrian; identifying abehavior of interest corresponding to an action taken by the objectbased on (1) the object data, (2) map information identifying a roadelement corresponding to the location of the object, (3) a comparisonbetween the road element and movement information for the object, thecomparison indicating that the object has performed the behavior ofinterest, and (4) the classification of the object; and maneuvering thevehicle, by the one or more processors, based on the identified behaviorof interest, wherein the road element comprises a first lane of trafficor an intersection.
 2. The method of claim 1, further comprising:retrieving from memory, a behavior model for objects configured topredict how another object, having a same classification as the object,will behave based on the classification; and maneuvering the vehicle, bythe one or more processors, based on the behavior model and theidentified behavior of interest.
 3. The method of claim 2, wherein thebehavior model indicates a probability that the object makes aright-hand turn at the intersection.
 4. The method of claim 2, whereinthe behavior model indicates a probability that the object makes aleft-hand turn at the intersection.
 5. The method of claim 2, whereinthe behavior model indicates a probability that the object goes throughthe intersection without stopping.
 6. The method of claim 2, wherein thebehavior model indicates a probability that the object stops at theintersection.
 7. The method of claim 2, wherein the behavior modelindicates an amount of time the object is expected to stop at theintersection or an amount of time the object is expected to move throughthe intersection.
 8. The method of claim 1, wherein the classificationcorresponds to a road vehicle or a pedestrian.
 9. A system formaneuvering a vehicle, the system comprising one or more processorsconfigured to: receive object data, wherein the object data comprises alocation of an object in an environment of the vehicle and movementinformation for the object; determine, from the object data, aclassification of the object, the classification being a type of avehicle or a pedestrian; identify a behavior of interest correspondingto an action taken by the object based on (1) the object data, (2) mapinformation identifying a road element corresponding to the location ofthe object, (3) a comparison between the road element and movementinformation for the object, the comparison indicating that the objecthas performed the behavior of interest, and (4) the classification ofthe object; and maneuver the vehicle based on the identified behavior ofinterest, wherein the road element comprises a first lane of traffic oran intersection.
 10. The system of claim 9, wherein the one or moreprocessors are further configured to: retrieve from memory, a behaviormodel for objects configured to predict how another object, having asame classification as the object, will behave based on theclassification; and maneuver the vehicle based on the behavior model andthe identified behavior of interest.
 11. The system of claim 10, furthercomprising the memory, and wherein the memory further stores the mapinformation.
 12. The system of claim 10, wherein the behavior modelindicates a probability that the object makes a right-hand turn at theintersection.
 13. The system of claim 10, wherein the behavior modelindicates a probability that the object makes a left-hand turn at theintersection.
 14. The system of claim 10, wherein the behavior modelindicates a probability that the object goes through the intersectionwithout stopping.
 15. The system of claim 10, wherein the behavior modelindicates a probability that the object stops at the intersection. 16.The system of claim 10, wherein the behavior model indicates an amountof time the object is expected to stop at the intersection or an amountof time the object is expected to move through the intersection.
 17. Thesystem of claim 9, wherein the classification corresponds to a roadvehicle or a pedestrian.
 18. The system of claim 9, further comprisingthe vehicle.
 19. A non-transitory, computer readable recording medium onwhich instructions are stored, the instructions, when executed by one ormore processors, cause the one or more processors to perform a method ofmaneuvering a vehicle, the method comprising: receiving object data,wherein the object data comprises a location of an object in anenvironment of the vehicle and movement information for the object;determining, from the object data, a classification of the object, theclassification being a type of a vehicle or a pedestrian; identifying abehavior of interest corresponding to an action taken by the objectbased on (1) the object data, (2) map information identifying a roadelement corresponding to the location of the object, (3) a comparisonbetween the road element and movement information for the object, thecomparison indicating that the object has performed the behavior ofinterest, and (4) the classification of the object; and maneuvering thevehicle based on the identified behavior of interest, wherein the roadelement comprises a first lane of traffic or an intersection.
 20. Themedium of claim 19, wherein the method further comprises: retrievingfrom memory, a behavior model for objects configured to predict howanother object, having the same classification as the object, willbehave based on the classification; and maneuvering the vehicle, basedon the behavior model and the identified behavior of interest.